Online ranking metric

ABSTRACT

Methods, systems, and apparatus, including computer program products, for ranking resources. One or more signals reporting one or more actions by the plurality of users related to a respective resource are received, where a reported action is a presentation of the respective resource, an addition of the respective resource, or a deletion of the respective resource. The reported actions include one or more short deletions of the respective resource. A respective long-addition click-through rate (LACTR) is determined for the respective resource based on a number of reported presentations of the respective resource and a difference between a number of reported additions of the respective resource and a number of short deletions of the respective resource. The plurality of resources are ranked based on the respective LACTRs for the plurality of resources. The ranking of the plurality of resources are provided.

RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §119(e) of U.S.Patent Application No. 60/862,003, titled “RANKING AND RECOMMENDATIONSERVER FOR BROWSABLE DIRECTORIES,” filed Oct. 18, 2006, which isincorporated by reference herein in its entirety:

This application is related to the following applications, which areincorporated by reference herein in their entirety:

U.S. patent application Ser. No. 11/874,888, titled “Generic OnlineRanking System and Method Suitable for Syndication,” filed Oct. 18,2007; and

U.S. patent application Ser. No. 11/874,889, titled “Online RankingProtocol,” filed Oct. 18, 2007.

BACKGROUND

This specification relates generally to the ranking of digitalresources.

“Browse-and-search” directories of content are proliferating on theWorld Wide Web. Examples include directories of 3D models for viewing ingeographic information applications, directories of widget applications,and directories of visual content such as images or videos. Thesedirectories allow users to find the content in which they are interestedthrough searching or through browsing by category.

The content in a browse-and-search directory can be ranked and presentedin the directory based on the ranking order. By presenting the contentin the ranking order, users viewing the directory can be presented withthe “good” content first. Content providers have an incentive to providecontent that are more likely to be ranked as “good,” in order toincrease the potential audience for their content. Manybrowse-and-search directories, however, do not have enough traffic oruser activity to justify the cost of having dedicated computingresources for determining rankings or having ranking capabilities on parwith the ranking capabilities implemented for web search oradvertisements.

SUMMARY

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions of,for each of a plurality of resources presented to a plurality of usersby a data processing service: receiving one or more signals reportingone or more actions by the plurality of users related to the respectiveresource, where a reported action is a presentation of the respectiveresource, an addition of the respective resource, or a deletion of therespective resource, the reported actions include one or more shortdeletions of the respective resource, a short deletion of the respectiveresource being a deletion of the respective resource occurring within apredefined time period after a corresponding addition of the respectiveresource, and determining a respective long-addition click-through rate(LACTR) for the respective resource based on a number of reportedpresentations of the respective resource and a difference between anumber of reported additions of the respective resource and a number ofshort deletions of the respective resource; ranking the plurality ofresources based on the respective LACTRs for the plurality of resources;and providing the ranking of the plurality of resources. Otherimplementations of this aspect include corresponding systems, apparatus,and computer program products.

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. A ranking service can be syndicated to multiple subscribers.The ranking service and the subscribers exchange data using a predefinedprotocol. The ranking service can be a generic service that is notcompetitive with ranking capabilities for web search. The rankingservice scores and ranks the resources of a subscriber using user actiondata that is reported by the subscriber. The resources of a subscribercan be scored and ranked by context. A subscriber can define thecontexts for which rankings of the resources of the subscriber are to begenerated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a ranking service and one or moresubscribers.

FIG. 2 is a block diagram of a ranking server, a subscriber server, anda client device.

FIG. 3 is a flow diagram of an example process for ranking resources.

FIG. 4 is a diagram illustrating an example exchange of data between aranking service and a subscriber.

FIGS. 5A thru 5C illustrate an example user interface of an examplepersonalized homepage.

FIG. 6 is a flow diagram of an example process for determining rankingsof resources.

FIG. 7 is a flow diagram of an example process for determining rankingsof resources by context.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating a ranking service 102 and one ormore subscribers 104. The ranking service 102 provides ranking servicesto the subscribers 104, as will be described. The ranking service may beimplemented on a computer system made up of one or more computers, whichmay be in a single location or geographically distributed. Users 106,using client computers or other client devices, may access any of thesubscribers 104. Users may be people or computer programs running in acomputer system or a device.

Each of the subscribers 104 to the ranking services is a data processingservice or system. Examples of data processing services or systemsinclude, without limitation, a personalized homepage service, a shoppingwebsite, a content sharing or distribution website or service, or asoftware download website or service. A subscriber 104 presents one ormore resources to users who access the subscriber, either by making theresources available directly to users or by presenting links to theresources. A resource, as used in this specification, is digital datathat has an identity and, in general, is retrievable from a repositoryor service using the identity or through some data processingoperations. A resource can include other resources. Examples ofresources include, without limitation, data representations of realitems for sale on a shopping website; software or data files; images,videos or audio clips; or widgets, gadgets, or modules in a personalizedhomepage service. A subscriber 104 can present its resources to users inan application. For example, the resources may be presented in a webpagerendered in a web browser application.

Users 106 can access a subscriber 104 and interact with the subscriber104 and/or with the resources provided by the subscriber 104. Forexample, a user can access a shopping website and view datarepresentations of items sold by the shopping website and place ordersfor the items. As another example, a user may access a personalizedhomepage service, view available widgets, gadgets, or modules that canbe added to a personal homepage, and add any number of the availablewidgets, gadgets, or modules to the personal homepage. In someimplementations, the resources of a subscriber are presented to theusers as a directory of resources (a “browsable directory”) throughwhich the users can browse and search. The top level of the browsabledirectory includes the totality of the resources that the subscribermakes available. The resources can be organized into categories andsub-categories, which are placed as sub-levels of the top-levelbrowsable directory.

Users 106 may access a subscriber 104 from any of a variety of clientcomputers or devices, e.g., desktop computers, notebook computers,tablet computers, network terminals, personal digital assistants (PDAs),mobile phones, smartphones, music or multimedia player devices, and soforth.

A ranking service 102 provides ranking services to subscribers 104. Inproviding ranking services, the ranking service 102 receives signalsfrom the subscriber, identifies from the received signals user actionsrelated to resources presented by the subscriber, and scores theresources based on the identified user actions. The resources are rankedbased on the scores and the ranking is provided to the subscriber or,optionally, to another requester (which may be another data processingservice), on request. Optionally, the ranking service also provides thescores. The ranking service 102 can provide ranking services to multiplesubscribers.

In some implementations, the ranking service 102 keeps the informationreported in the signals indefinitely, allowing the ranking service 102to build long-term user action histories of users and resources.

The ranking service 102 can provide a ranking for a subscriber'sresources with respect to all of a subscriber's resources as a whole or,optionally, with respect to a subscriber-defined subset of thesubscriber's resources. Subsets can be based in whole or in part on oneor more of geographic boundaries, political boundaries, demographicinformation, time, language, industry, or access rights. The rankingservice 102 can optionally also provide rankings with respect to aparticular user. For example, a requester may request a ranking thattakes into account a user profile or a user action history of a user.

The score of a resource is a value that is a function of the useractivity related to the resource. A ranking is an ordered representationof resources that indicates an order determined by their respectivescores.

The ranking service 102 and the subscribers 104 communicate throughinterfaces 108 and 110. The communication between the ranking service102 and the subscribers 104 through the interfaces 108 and 110 includethe sending and receiving of data using one or more predefinedprotocols. That is, the ranking service 102 uses the same protocol tosend data to and receive data from multiple subscribers, and eachsubscriber uses the same protocol to send data to and receive data fromthe ranking service 102. In some implementations, different classes ofsubscribers are offered different protocols suitable for theirparticular requirements.

The ranking service 102 generates its scores and rankings of resourcesfor each subscriber based on data provided by the subscriber. Theranking service 102 can accept the data provided by a subscriber as is(assuming that the data is sent in accordance with the protocol) anddoes not attempt to acquire, for ranking purposes, data beyond what isprovided by the subscriber. For this reason, the ranking service 102 canaccept data from a subscriber that may have significance to thesubscriber but has no a priori significance to the ranking service.

Nevertheless, the ranking service can detect correlations in such dataand use the correlations in determining rankings.

In some implementations, the architecture of the ranking service 102 andthe subscribers 104 are implemented in accordance with therepresentational state transfer (REST) software architectural style.

Ranking Data Protocol

The ranking service and subscribers may send and receive data using aspecific protocol (generically, the “ranking data protocol,” forconvenience) that specifies the form and the significance of data sentbetween a ranking service and a subscriber. The protocol may bepublicized by the ranking service in any suitable manner, e.g.,publication in a website, e.g., in the form of a WSDL (Web ServicesDescription Language) document. Any subscriber may use the protocol.

The ranking data protocol also specifies how user actions are reported.The ranking data protocol specifies that a signal reporting a useraction identifies the user action and identifies a resource to which theuser action relates. In some implementations, the ranking data protocolspecifies that user actions are reported as one of three types: views,additions, and deletions.

A subscriber would report an action as a view of a resource when thesubscriber has presented or displayed the resource to the user. In anexemplary personalized homepage implementation, a view may be any actionin which a resource is displayed to a user. Whether the resource isdisplayed to a user in response to a search by a user or as a result ofbrowsing for resources by the user, the action of displaying may becounted as a view. In an online store implementation, the displaying ofinformation associated with a product to a user may be reported as aview of a resource (the resource being the data representation of theproduct, including the information associated with the product).Similarly, for a software download website implementation, thedisplaying of information associated with a software download may bereported as a view of a resource, namely a view of the software andrelated information.

A subscriber would report an action as an addition of a resource whenthe subscriber has received information that a resource has beenaffirmatively accepted by a user. In the example personalized homepageimplementation, an addition action would be reported when a user adds aresource to the user's personalized homepage. In the example shoppingwebsite implementation, an addition of a resource, i.e., an item sold bythe shopping website, would be reported when a purchase of the itemoccurs. In the example software download website implementation, anaddition of a resource, i.e., a piece of software available for downloadfrom the website, would be reported when a download of the piece ofsoftware by the user to a client machine occurs.

A subscriber would report an action as a deletion of a resource when thesubscriber has received information that a resource has beenaffirmatively rejected by a user after an addition of the resource bythe user. In the example personalized homepage implementation, adeletion action would be reported when a user deletes from apersonalized homepage a resource the user had previously added to thehomepage.

Generally, each individual subscriber decides what is and what is not aview, addition, or deletion. That is, each subscriber has its owndefinition of what actions are reported as a view, as an addition, or asa deletion to the ranking service.

In some implementations, the types of reportable actions in the rankingdata protocol go beyond views, additions, and deletions. Inaction oridleness related to a resource can also be reported. For example, if aresource has not been added by any user during some period of time, thislack of usage activity for a resource may be reported to the rankingservice. The ranking service may take this lack of usage activity intoaccount when scoring the resource.

In the implementations where the user actions are reported as views,additions, and deletions, the score of a resource may be calculatedbased on the views, additions, and deletions related to the particularresource. In some implementations, the manner in which the score of aresource is calculated is predefined. In some other implementations,various scoring functions may be made available. The subscriber orrequester may specify which function to use. If a scoring function isnot specified by the subscriber or requester, the ranking service mayuse a default scoring function. For example, one subscriber may want itsresources scored and ranked based only on views, while anothersubscriber may want its resources scored and ranked based on views andadditions.

In some implementations, a score for the resource is calculated based onso-called “long additions.” A “long addition” is an addition of theresource by the user, followed by a deletion of the resource by the sameuser more than a predefined time period after the addition or followedby no deletion of the resource by the same user. In someimplementations, the number long additions for a resource is calculatedas follows:number of long additions for a resource=number of additions for theresource−number of short deletions for the resource,where a short deletion is a deletion of the resource by a user withinthe predefined time period after the addition of the resource by theuser. In some implementations, the predefined time period for a deletionto be considered as a short deletion differs depending on the type ofresource. For example, the predefined time period for a video may be 10seconds, but may be 1 hour for a widget or gadget in a personalized homepage service. In some implementations, whether a deletion is a shortdeletion or otherwise is up to the subscriber to determine and report tothe ranking service.

In some implementations, a long addition can be defined by a subscriberin other ways depending on the type of resource. For example, for avideo, a long addition can be defined as an instance of completion (ornear-completion) of playback of the video for its entire duration.

The score for the resource can be the number of long additions for theresource, some function of the number of long additions for theresource, or some function of the number of long additions for theresource and of other values, e.g., the number of views for theresource. An example of a scoring function that uses the number of longadditions is described below in reference to FIGS. 6-7. In someimplementations, a decay function can be applied to the counts of theactions. For example, the number of views for a resource can decaylinearly or otherwise over time. The decay can have the effect ofdeemphasizing the influence of actions that occurred a relatively longtime ago on the rankings.

In some implementations, the signals reporting user actions also includeindications of the times at which the reported actions occurred. Forexample, the subscriber can report additions and deletions, with theirrespective times. The times can be used to determine if a deletion is ashort deletion. In some implementations, the signals report specificallywhether a deletion is a short deletion; in such implementations, thesubscriber determines whether a deletion is a short deletion and reportsshort deletions as such.

The ranking data protocol need not specify a naming convention or anamespace for resources. A subscriber can define its own namingconvention and/or namespace for its resources. The naming convention ornamespace defined by a subscriber may use arbitrary names for resources.The resource names defined by a subscriber have no a priori significanceto the ranking service. The naming convention or namespace defined bythe subscriber may include the use of pseudonyms, code names, or thelike. The ranking service associates user actions related to a resourcewith whatever name is provided for that resource by the subscriber. Whenthe ranking service provides scores or rankings to a requester andincludes names of resources in the scores or rankings, the includednames are the pseudonyms, code names, or whatever names that areprovided by the subscriber. Identification of the resourcescorresponding to the pseudonyms or coded names is left to the subscriberor requester. The protocol optionally includes an optional parameterwith which a subscriber can identify a namespace for its resources.

For some subscribers, names for resources may be relatively long. Forexample, Universal Resource Locators (URLs) of resources may be used asresource names. The URL for a resource may include various parametersand as a result become relatively long. In some implementations, ratherthan sending the full URL for a resource as the name of the resource, asubscriber generates a shorter name for the resource by, for example,encoding the URL of the resource and then generating a hash of theencoding. Other encoding or name mapping schemes may be used. Theshorter, encoded name can then be used in signals reporting useractions. The shorter, encoded name can be used to provide anonymity forthe resources and/or to optimize performance by reducing the amount ofdata transmitted and processed.

The signals can include one or more parameters and values for theseparameters. Such parameters can be used to qualify the user actionsreported in the signals and give the user actions context. The contextsmay be used to generate finer rankings for resources or to generatenormalized rankings. For example, if the resources are modules forpersonalized homepages and the modules are made available topersonalized homepages in various languages, the modules may have ascore and/or ranking for each of the various languages, as well as anoverall score/ranking. A signal reporting a user action related to anyof the modules will include a parameter indicating the language contextin which the user action occurred.

Signals from a subscriber may include an identifier of the subscribersending the signals. A subscriber may use an arbitrary string toidentify itself in its signals. To ensure that subscriber names areunique, the ranking service may append additional characters tosubscriber names. In some implementations, the subscriber may alsoidentify a subdivision of the subscriber with which the user action isassociated. For example, in an example module service subscriber thatprovides modules for addition to personalized homepages and desktopsearch application environments, the module service subscriber mayidentify whether a module was added to or deleted from a personalizedhomepage or a desktop search application environment. A ranking servicemay generate rankings with respect to particular subdivisions of asubscriber.

In some implementations, the ranking service does not require that asubscriber register with the ranking service before the subscriber sendsdata to the ranking service. In some other implementations, thesubscriber is required to register by providing information identifyingthe subscriber, for example, and receiving a subscriber identifier andoptionally a password or cookie. The ranking service may enforceuniqueness of subscriber identifiers at the pre-registration stage byrequiring selection of a unique subscriber identifier duringregistration.

Signals from a subscriber may include, for a reported user action, anidentifier of the user who initiated or performed the action. The namingconvention for users is left to each subscriber. By including the useridentifier in the signal, the ranking service receiving the signal canidentify resources that the user has viewed or added, and group thisuser with other users who have viewed or added similar resources.

The parameters and values included in signals may include parameters andvalues defined by the ranking service and parameters and values definedby the subscriber sending the signals. The ranking service may defineparameters that any subscriber can use in its signals. The parametersdefined by the ranking service would be those the services considers tobe of general significance for subscribers. The parameters and valuesdefined by a ranking service may be defined as part of the ranking dataprotocol for the ranking service.

In some implementations, the parameters defined by the ranking serviceas part of the ranking data protocol include language, geographicregion, and category. A language parameter and corresponding valuesindicate the language context in which a user action relating to aresource occurred. The language context may refer to the language of theresource itself or the language of the environment in which the actionrelated the resource occurred. For example, if the user added aFrench-language module to a personalized homepage, the action (theaddition) may be reported as having taken place in a context where thelanguage is French. As another example, a module may be added to anEnglish-language personalized homepage and a German-languagepersonalized homepage. The signal reporting the addition to the Englishhomepage may include a value specifying “English” for the languageparameter, and the signal reporting the addition to the German homepagemay include a value specifying “German” for the language parameter. Insome implementations, a signal may include a value for the languageparameter in the format “lang=value,” where the value indicates thelanguage. Values for the language parameter may be specified usingwell-known language codes, e.g., the ISO (International Organization forStandardization) 639 set of language codes.

A geographic region or locale parameter may also be defined to givecontext to user actions. The region parameter indicates the location ofthe user or the location of the action. For example, if a user action isdetermined to have taken place in Canada (e.g., because the action wasperformed at a subscriber in the “.ca” country code top-level domain),the signal reporting the action may include a value specifying “Canada”for the region or locale parameter. In some implementations, “geographicregion” or “locale” are countries or nations. Values for the geographicregion or locale parameter may be specified using well-known countrycodes, e.g., ISO 3166 country codes.

A topical category parameter may also be defined by the ranking service.Examples of topical categories include sports, politics, technology,lifestyle, and entertainment. Sub-categories may also be defined withincategories.

Parameters for a user identifier and a subscriber identifier may also bedefined by the ranking service. For these parameters, the value spacesfor these parameters may be defined by the individual subscriber.

In some implementations, the ranking service, when calculating scoresand rankings, may take into account whether a resource was presented atthe top or bottom of a page and/or whether the resource was presented inthe first page or a subsequent page within multiple pages of resources.A parameter for specifying the position in which, or, more generally,the prominence with which, a resource was presented may be defined bythe ranking service for use by subscribers.

A subscriber may also define one or more parameters and correspondingvalues. Subscriber-defined parameters and corresponding values willpresumably be significant to the defining subscriber but will notnecessarily have any meaning to other subscribers or to the rankingservice. As with names for resources, the ranking service attributes noa priori significance to subscriber-defined parameters and acceptssubscriber-defined parameters and values as they are provided by thesubscriber. Indeed, two subscribers may use the same text string tospecify respective parameters and the ranking service will treat theparameters as unrelated to each other because they are associated withdifferent subscribers.

Examples of subscriber-defined parameters could include, for example,clothing size defined by a subscriber that is a shopping website, orlicense type (freeware, shareware, and so on) for a subscriber that is asoftware download website.

In some implementations, a signal need not include values for all of theparameters defined by the ranking service or all of thesubscriber-defined parameters. The ranking service associates an actionreported in the signal with whatever context parameter values arespecified. If a signal reporting an action does not specify any contextparameter values, the action reported in the signal may be associatedwith a default set of context parameter values.

In some implementations, a signal can be an aggregated signal reportingactions for multiple resources. For example, a signal may report thatresources A, B, C, and D were viewed by the same user. The number ofresources for which actions are reported in a signal or the number ofactions reported in the signal may be capped. For example, a signal mayreport actions for up to a predefined number of resources.

The ranking data protocol also defines how rankings (and optionally,scores) are requested and reported. In some implementations, a requestfor rankings or scores includes an identifier of the subscriberassociated with the resources for which rankings or listings arerequested, optional context parameters such as language, category orsubscriber-defined parameters, and optional values, if supported by theranking service, indicating how many resources to include in therankings and which resources to include. The identifier of thesubscriber identifies the subscriber associated with the resources forwhich scores or rankings are requested. For example, if a personalizedhomepage service is requesting rankings for its resources, the homepageservice would identify itself in the request and optionally one or morepertinent subdivisions of the homepage service. The request may includeone or more context parameters; the context parameters indicate forwhich context are rankings being requested. For example, a requestspecifying that the language context is English and the geographicregion context is the United States may be understood as a request forrankings with respect to an English-United States context. Contextparameters may include the parameters defined by the ranking service,e.g., language, geographic region, and category, as well as anysubscriber-defined parameters.

In some implementations, a request can indicate how many resources toreturn, a starting offset, or both. For example, a request may indicatethat 30 resources, starting from the 11th highest ranked resource, bereturned. This can be used for displaying the ranked resources to a userover multiple pages. For example, if 10 resources can be displayed perpage, the first 10 ranked resources can be requested for display in thefirst page, the second 10 ranked resources can be requested for displayin the second page, and so on.

The ranking service may report the rankings as a listing of resourceslisted in their ranking order. The reported rankings identify theresources using the resource names as provided by the subscriber.

The ranking data protocol may be implemented using any of a variety oftechnologies for exchanging data or messages, including but not limitedto Simple Object Access Protocol (SOAP), Extensible Markup Language(XML), Hypertext Transport Protocol (HTTP), and HTTP+XML. For example,the signals can be transmitted from a subscriber using XML or HTTP POSTmethods.

In some implementations, data is exchanged between the ranking serviceand a subscriber in Uniform Resource Locators (URL) or in UniformResource Identifiers (URI) sent in HTTP messages. That is, the signals,rankings requests, and reported rankings are all strings embedded inURLs or URIs. For example, a signal URL reporting actions might looklike:

-   -   http://www.rankingserver.com/idofsubscriber/roa?action=v&subs=nameofsubscriber&ob        js=aZ,t,f,g2D,31R,42,N3rD        where “roa” stands for “report of actions” and “roa?” indicates        that this is a signal reporting user actions. The “action”        parameter indicates the action being reported (here, “v” for        view). The “subs” parameter is a name of the subscriber. The        “objs” parameter is a list of resources affected by the action        (viewed, added, deleted, and so on). Thus, the example signal        URL above reports views of resources “aZ,” “t,” “f” “g2D,”        “31R,” “42,” and “N3rD.”

A signal URL that reports an addition might look like this:

-   -   http://www.rankingserver.com/idofsubscriber/roa?action=a&subs=nameofsubscriber&ch        anged=g2D&objs=aZ,t,f,g2D,31R,42,N3rD        where the value “a” for the “action” parameter indicates an        addition being reported, and the value “g2D” for the “changed”        parameter identifies the added resource. Thus, this sample URL        reports an addition of resource “g2D” and views of views of        resources “aZ,” “t,” “f,” “g2D,” “31R,” “42,” and “N3rD.” Other        parameters may be included in the URL as appropriate.

A request URL requesting rankings may look like this:

-   -   http://www.rankingserver.com/idofsubscriber/objs?req=blogger&num=10&start=10

In this example, “objs?” indicates that this URL is a request forrankings. The “req” parameter is a name of the requester. The numparameter indicates how many items to return. In some implementations,the entire result set of ranked resource is downloaded if num=0. Thestart parameter indicates a starting offset in the set of rankedresources. For example, a request for num=10 and start=1 requests thatthe 10 highest ranked resources (10 resources starting from the firsthighest ranked) be returned. A request for num=10 and start=11 requeststhe next 10 highest ranked resources (10 resources starting from theeleventh highest ranked).

In some implementations, the ranked resources reported by the rankingservice are reported as text strings of their names, listed in theirranking order, with optional separator characters between each namestring.

FIG. 2 is a block diagram of a ranking server 202, a subscriber server210, and a client device 220. The ranking server 202 includes a rankingengine 204 and a recommendation engine 206. The ranking server 202 alsoincludes resource ranking data 208, which includes data from signalssent by subscriber server 210 and ranking data generated by the rankingserver 202. The data from the signals may be kept indefinitely (i.e.,not deleted even when the data has aged beyond a certain amount) or not.In some implementations, the resource ranking data 208 may be stored ina database.

The ranking engine 204 generates scores and rankings for resourcesprovided by subscribers. The recommendation engine 206 receives requestsfor rankings. The recommendation engine 206 provides rankings inresponse to the requests. In some implementations, the recommendationengine 206 may provide rankings that take into account informationregarding a particular user.

The subscriber server 210 includes a resources server 212, a signalsprovider 214, a ranking data requester 216, and resources 218. Theresources server 212 serves resources 218 to client devices 220. Thesignals provider 214 gathers user action data and sends the user actiondata to the ranking server 202. The ranking data requester 216 sendsrequests for rankings from the ranking server 202.

The ranking server 202 and the subscriber server 210 include aninterface 228 and 230, respectively. The ranking server 202 and thesubscriber server 210 send and receive data through the respectiveinterfaces. In some implementations, interfaces 228 and 230 implement aprotocol that defines the format for communications between the rankingserver 202 and subscriber server 210.

The client computer 202 includes a client application 222 and a useraction data collector 224. A user may access a subscriber through aclient application 222. In some implementations, a user accesses thesubscriber through a web-based interface, and the client application isa web browser that, among other things, renders and displays theweb-based interface. A user action data collector 224 monitors useractions related to resources presented by subscribers, and forwards datacharacterizing the user actions to the pertinent subscriber.

In some implementations, a user action data collector 224 may bespecific to a particular subscriber, in which case, the user action datacollector 224 collects user action data for only that subscriber. Insome other implementation, a user action data collector 224 may collectuser action data for multiple subscribers. In some implementations, theuser action data collector 224 is a JavaScript module embedded in theweb-based interface of a subscriber. When a client device 220 accessesthe web-based interface, the JavaScript module may be downloaded to theclient device 220 and activated when the web-based interface is renderedby the client application 222. In some other implementations, the useraction data collector 224 may be a standalone application, a modulewithin a standalone application, or a module within an add-on or plug-inapplication. For example, the user action data collector 224 may be amodule in a toolbar add-on to a web browser application. A client devicemay include multiple user action data collectors.

In some other implementations, a user action data collector 224 may sendthe user action data to a user action database. The signals provider 214may access the user action database and retrieve the data for sending tothe ranking server 202.

In some implementations, the collection of user action data takes placeat the client device 220 and at the subscriber server 210. User actionsare monitored at the client device 220 and data regarding user actionsare sent from the client device 220 to the subscriber server 210 foraggregation. In some other implementations, the collection of useraction data takes place entirely at the subscriber server 210. Thesubscriber server 210 (e.g., the signals provider 214) may, in thenormal course of processing inputs from clients devices 220, monitor andcollect the user action data associated with user actions related to theresources. The one or more networks 226-A or 226-B may include localarea networks, wide area networks, wired or wireless networks, and theInternet. In some implementations, the networks 226-A and 226-B are thesame (e.g., the Internet).

FIG. 3 is a flow diagram of a process of ranking resources. Whileprocess flow 300 illustrates the process with respect to one subscriber,the process flow 300 is applicable to each of multiple other subscribersas well.

Signals are received from a subscriber (302). The signals from thesubscriber include information characterizing user actions related toresources presented by the subscriber. The signals also include valuesfor one or more predefined parameters and values for one or moresubscriber-defined parameters. The subscriber-defined parameters arespecific to the subscriber. In some implementations, the signals alsoinclude identifiers of the users who performed the user actions reportedin the signals.

A ranking service may receive signals from multiple subscribers. Signalsreceived from a first subscriber include information characterizing useractions related to individual ones of first resources presented by thefirst subscriber. Signals received from a second subscriber includeinformation characterizing user actions related to individual ones ofthe second resources presented by the second subscriber. The signalsreceived from the first subscriber may also include values for one ormore parameters that are defined by the ranking service and values forone or more parameters defined by and specific to the first subscriber.The signals received from the second subscriber may also include valuesfor one or more parameters that are predefined by the ranking serviceand values for one or more parameters defined by the second subscriberand specific to the second subscriber.

Ranking data are generated from the signals (304). Ranking data, whichranks the resources, are generated from the received signals, values inthe signals for parameters specific to the subscriber, and values in thesignals for predefined parameters. Ranking data for the subscriber,however, are not generated based on values for parameters specific toother subscribers. For example, ranking data for resources presented bythe first subscriber may be generated from the first signals, values inthe signals for parameters defined by the ranking service, and values inthe signals for parameters defined by the first subscriber, but not fromvalues for parameters defined by any other subscriber. Similarly,ranking data for resources presented by the second subscriber may begenerated from the second signals, values in the signals for parametersdefined by the ranking service, and values in the signals for parametersdefined by the second subscriber, but not from values for parametersdefined by any other subscriber.

Generation of ranking data includes scoring resources and ranking theresources based on the scores. Example ranking functions are describedbelow in reference to FIGS. 6-7.

Optionally, the ranking service supports a plug-in ranking algorithm,which enables a syndicator to provide its own ranking function based onthe signals and implemented in agreed-upon language, e.g., Perl, Python,or the like. In some implementations, the ranking function is providedas an optional parameter with the request to return the rankedresources, e.g., &rank_func=<url to the code for ranking>. The firstsuch request may be computationally expensive because existing resourcesmay all have to be re-ranked. On subsequent requests with the sameranking function, however, the required computation would besignificantly less because the ranking server can cache the rankingresults. If the ranking server detects that a given &rank_func value ispopular, the ranking server can prefetch this code and pre-compute therankings to provide good response to an initial ranking request. Inaddition, the ranking server can disable the cache, e.g., to enabletesting new functions; and the ranking server can delete cached rankingsif it runs out of storage space. In some implementations, the subscribercan specify a requested amount of time to keep the cached data.

In some implementations, the ranking of resources is query-independent.That is, the ranking function does not take into account a query forresources that a user may have submitted to a subscriber. In suchimplementations, a resource is ranked based on its user action activity,and the relevance of a resource to a query is of no consequence. In someother implementations, the ranking of resources is query-dependent. Thatis, the ranking function takes into account a query for resources that auser submits. The query with respect to which resources are to be rankedis provided to the ranking service by the subscriber, as an additionalparameter, for example. In such implementations, a resource is rankedbased on some measure of relevance to the query or a combination ofrelevance to the query and user action activity. For example, a resourcethat satisfies a query is ranked higher than a resource that does notsatisfy the query, all else being equal. In some implementations, asubscriber can specify whether it wants a query-independent or aquery-dependent ranking. If the subscriber specifies its own rankingfunction, it can specify a query-independent or a query-dependentranking function.

FIG. 4 is a diagram illustrating an exchange of data between a rankingservice and a subscriber. The data exchange 400 illustrates the datacommunications between a subscriber and the ranking service.

The subscriber receives user actions related to resources presented bythe subscriber (402). The subscriber reports the user actions to theranking service (404). The user actions may be reported in the signalsdescribed above. In some implementations, signals may be sentperiodically by the subscriber to the ranking service. In some otherimplementations, signals may be sent as the user actions are received bythe subscriber or at other rates.

The ranking service generates rankings of resources for this subscriberbased on the actions (406). The rankings are generated based on theactions as reported by the subscriber (e.g., in signals sent by thesubscriber). In some implementations, rankings are generatedperiodically. In some other implementations, rankings are also generatedin response to requests for rankings. That is, when a ranking request isreceived, the rankings that are responsive to the request arere-generated so that the rankings sent in response to the request arefresh.

The subscriber (or another requester) may request ranking data from theranking service (408). The ranking service returns ranking data to therequesting subscriber or requester (410).

The rankings includes a list of resources in their ranking order. Whenthe subscriber presents the resources included in the rankings to auser, the subscriber may display the resources in the ranking order orin some other order. Furthermore, if the rankings were requested by thesubscriber in connection with a search for resources, any resources inthe reported rankings that do not match the search criteria may beomitted by the subscriber from the resources displayed in response tothe search.

As described above, rankings for a resource are generated based onsignals reporting user actions for the resource. If a resource has noreported action, a ranking cannot be generated for the resource. Asubscriber can provide signals reporting fabricated user actions as seeddata for its resource rankings. The ranking service can accept the seeddata as they are and generate initial rankings for the relevantresources using the seed data.

In some implementations, the ranking service identifies some useractions reported by a subscriber as not trustworthy. For example, thesubscriber can be the victim of attack, which can lead to the subscriberreporting a statistically unlikely burst of activity. The rankingservice can identify these reported actions as suspicious and ignorethem in the ranking.

When a user performs actions related to resources, a history of useractions accumulates in the ranking service. Based on that history,particular characteristics or contexts may be associated with the user.For example, a user may be associated with a group of users (or“recommendation group”) associated with a context where the language isFrench and the region is Canada. The user is associated with theFrench-Canada recommendation group because the user has viewed and/oradded resources that are popular among other users in the French-Canadarecommendation group. That is, the user has viewed and/or addedresources that are popular within the French-Canada context. When theuser accesses a subscriber, the subscriber may, recognizing the user anddetermining that the user is associated with the French-Canadarecommendation group, request for this user resource rankings withrespect to the French-Canada context.

As described above, rankings of a resource with respect to variouscontexts may be generated. Such rankings are sub-rankings of an overallranking for a resource, the overall ranking taking into account allcontexts. It may be the case that there may not be sufficient useraction activity for some contexts to generate sufficiently reliablerankings for such contexts. For example, the level of user actionactivity in a French-China context may be much less than the level ofuser action activity in the French-Canada or French-France contexts. Insuch circumstances, when a user is accessing a subscriber under theFrench-China context, the ranking service may “blend” contexts together.

In some implementations, the blending is done by, in effect, aggregatingsignals associated with multiple contexts when generating rankings. Forexample, if rankings for the French-China context are desired, theranking service may take into account, when generating rankings, useractions from other contexts where the language is French. In some otherimplementations, the ranking service may generate and report rankingsfor a higher level (i.e., broader) context, e.g., as reporting therankings for the French context when rankings for the French-Chinacontext are requested. In further implementations, the ranking servicemay drop certain contexts. For example, if a user action was reported tohave been performed in the English-Estonia context, the ranking servicemay drop the Estonia region context.

The blending of contexts may occur when a blending condition issatisfied. In some implementations, the blending condition is when thelevel of user action activity in a particular context is below a minimumthreshold.

In some implementations, blending may also occur across subscribers.Because the ranking service generally treats resources from differentsubscribers as unrelated, subscribers that wish to enable cross-blendingneed to use a common namespace for their resources and identifythemselves as agreeing to have their data used to determine rankings forother subscribers. The ranking data protocol specifies how subscriberscan identify themselves as willing to share data.

FIGS. 5A thru 5C illustrate an example user interface of an examplepersonalized homepage. A personalized homepage service may providepersonalized homepage spaces to users and resources for presentation ina personalized homepage space. A user may select the resources to bepresented in the user's personalized homepage. In other words, the userhas some control over what resources go into their personalizedhomepage.

In some implementations, resources that may be added to a personalizedhomepage include widgets, gadgets, or modules. Such resources performparticular tasks or display particular content. For example, a modulemay display a television schedule grid or a calendar. As anotherexample, a module may display the content of a Really Simple Syndication(RSS) feed. In some implementation, such resources may be implemented asJavaScript modules that may be embedded in the personalized homepage.The resources are activated when the personalized homepage is renderedin a web browser application.

A personalized homepage 500 may include any number of resources (shownas resources 504-508 in FIG. 5A). The personalized homepage 500 may alsoinclude a link 502 or some other manner of navigating to a customizationpage 510, where the user may browse and/or search for resources to addto the personalized homepage.

The customization page 510 may include a link 512 back to thepersonalized homepage 500. The customization page 510 presents one ormore available resources 516 or information about them. If the number ofresources to be presented exceeds the display region capacity of thepage, the customization page 510 may include “previous” and “next” links514 for navigation between pages of resources. Each resource displayedin the customization page or pages that the user has navigated to may bereported by the personalized homepage service to the ranking service asviewed by the user.

The resources 516 may be displayed in the customization page 510 in anorder that is based, at least in part, on the rankings of the respectiveresources 516 as reported by a ranking service. In some implementations,other considerations may affect the order in which the resources aredisplayed. For example, in some implementations, the top 30 resourcesmay be presented in a randomized order, in order to reduce the effect ofthe bias towards higher-ranked resources that may occur whenhigher-ranked resources are always displayed first.

In some implementations, the customization page 510 allows a user tobrowse available resources or search for resources meeting specifiedcriteria. The customization page 510 may include one or more searchfields for searching for resources, or a link to a search page may beincluded in the customization page 510 for navigating to the searchpage. The search fields may allow for searching by keyword, author,date, category, or any other suitable criterion or combination ofcriteria.

As an example scenario, consider a user who wishes to add Resource A 518to a personalized homepage 500. The user can select the “Add” button 520in the customization page 510. Then, in the personalized homepage 500,Resource A 518 is presented along with resources 504-508. Thepersonalized homepage service reports the adding of Resource A to theranking service as an addition action related to Resource A.

The user can also delete a resource from the personalized homepage 500.For example, if the user wishes to delete Resource Z 508, the user mayclick on the “X” 509 right above the resource. After the user deletesResource Z 508, Resource Z 508 is no longer displayed in thepersonalized homepage 500 until the user adds it back. The personalizedhomepage service reports the deletion of Resource Z to the rankingservice as a deletion action related to Resource Z.

FIG. 6 is a flow diagram of an example process 600 for determiningrankings for resources. Resources are ranked periodically (e.g., dailyor hourly). The resources are ranked by determining a long add clickthrough rate (LACTR) for the resources. The LACTR of a resource is ascore that is used to rank the resource.

When it is time to determine rankings for resources (602), a LACTR iscalculated for each of the resources in accordance with the receivedsignals reporting actions for the resources (604). A LACTR is calculatedfor each resource available to a user (606, 608). In this example, thecalculations are performed in a loop (604, 610).

A number of long additions is calculated for a resource (606). Thenumber of long additions is the number of additions for the resourceminus the number of short deletions for the resource, as describedabove.

A LACTR is calculated for the resource (608). The LACTR for a resourceis the number of long addition for the resource divided by the number oftimes the resources was displayed. In some implementations, the numberof views for the resource is used as the number of times the resourcewas displayed.

The loop ends when a LACTR is calculated for each of the resources(610). The resources are ranked by their LACTRs (612), i.e., by orderingthe resources according to their LACTRs. The higher the LACTR of aresource, the higher the resource is ranked. In some implementations,not all of the resources for which a LACTR was calculated is included inthe rankings. For example, a resource may be omitted from the rankingsif its LACTR does not meet or exceed a predefined threshold.

FIG. 7 is a flow diagram of an example process 700 for determiningrankings for resources by context. In some implementations, a LACTR iscalculated for a resource for each context (e.g., language, geographicregion, category, etc.) in which user actions related to the resourcewere reported. Resources are ranked within a context according to theirLACTRs for that context.

When it is time to determine rankings for resources (702), if actionswithin contexts have been reported for the resources, a LACTR iscalculated for each of the resources for one or more contexts inaccordance with the received signals reporting actions for the resourcesin the contexts (706, 708). A LACTR is calculated for each resource(loop 708, 714) for each of one or more contexts (loop 706, 718) andresources are ranked within each context.

A number of long additions within a context is calculated for a resource(710). The number of long additions in a context is the number ofadditions for the resource within the context minus the number of shortdeletions for the resource within the context.

A LACTR is calculated for the resource within the context (712). TheLACTR for a resource within the context is the number of long additionfor the resource within the context divided by the number of times theresources was displayed within the context. In some implementations, thenumber of views for the resource within the context is used as thenumber of times the resource was displayed. The within-context LACTRserves as a score for the resource within a context.

The loop for calculating within-context LACTRs for resources ends when awithin-context LACTR is calculated for the resources (714). Theresources are ranked within a context by their LACTRs within the context(716) by ordering the resources according to their LACTRs. The higherthe LACTR of a resource, the higher the resource is ranked. In someimplementations, not all of the resources for which a LACTR wascalculated are included in the rankings. For example, a resource may beomitted from the rankings within a context if its LACTR within thecontext does not meet or exceed a predefined threshold.

The loop of contexts ends when rankings for all of the one or morecontexts have been determined (718).

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be a propagated signal or acomputer-readable medium. The propagated signal is an artificiallygenerated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a computer.The computer-readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them.

The term “data processing apparatus” encompasses all apparatus, devices,and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub-programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described is this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the following claims. For example, the actions recited in theclaims can be performed in a different order and still achieve desirableresults. As one example, the processes depicted in the accompanyingfigures do not necessarily require the particular order shown, orsequential order, to achieve desirable results. In certainimplementations, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. A computer-implemented method, comprising: foreach of a plurality of resources presented to a plurality of users by adata processing service: receiving one or more signals originating fromthe data processing service reporting one or more actions by theplurality of users related to the respective resource, wherein areported action is a presentation of the respective resource, anaddition of the respective resource, or a deletion of the respectiveresource, wherein the reported actions comprise one or more shortdeletions of the respective resource, a short deletion of the respectiveresource being a deletion of the respective resource occurring within apredefined time period after a corresponding addition of the respectiveresource; determining a number of long additions for the respectiveresource by subtracting a number of short deletions of the respectiveresource from a number of reported additions of the respective resource;and determining a respective long-addition click-through rate (LACTR)for the respective resource by dividing the number of long additions forthe respective resource by a number of reported presentations of therespective resource; ranking the plurality of resources based on therespective LACTRs for the plurality of resources; and providing theranking of the plurality of resources to the data processing service,wherein receiving, determining, ranking, and providing are performed byone or more computers.
 2. The method of claim 1, wherein: providing theranking of the resources comprises providing representations of at leasta subset of the resources in ranking order.
 3. The method of claim 1,wherein: the one or more signals each specify a context in which therespective reported actions occur; and determining a respective LACTRfor the respective resource comprises determining a respective LACTR forthe respective resource in the context based on the reported actionsrelated to the respective resource in the context.
 4. The method ofclaim 3, wherein the context is one or more of language, geographicregion, or topical category.
 5. The method of claim 1, wherein the oneor more signals are embedded in one or more Universal Resource Locators(URLs).
 6. The method of claim 1, further comprising: identifying firstusers of the plurality of users, the first users associated with anaction history that indicates a preference for a first resource and asecond resource of the plurality of resources; identifying a second userof the plurality of users, the second users associated with an actionhistory that indicates a preference for the first resource; andmodifying a ranking of the second resource with respect to the seconduser based on the indicated preference for the first resource by thefirst and second users.
 7. A system, comprising: one or more processors;and instructions configured for execution by the one or more processors,the instructions comprising instructions to: for each of a plurality ofresources presented to a plurality of users by a data processingservice: receive one or more signals originating from the dataprocessing service reporting one or more actions by the plurality ofusers related to the respective resource, wherein a reported action is apresentation of the respective resource, an addition of the respectiveresource, or a deletion of the respective resource, wherein the reportedactions comprise one or more short deletions of the respective resource,a short deletion of the respective resource being a deletion of therespective resource occurring within a predefined time period after acorresponding addition of the respective resource; including: determinea number of long additions for the respective resource by subtracting anumber of short deletions of the respective resource from a number ofreported additions of the respective resource; and determine arespective long additional click-through rate (LACTR) for the respectiveresource by dividing the number of long additions for the respectiveresource by a number of reported presentations of the respectiveresource; rank the plurality of resources based on the respective LACTRsfor the plurality of resources; and provide the ranking of the pluralityof resources.
 8. The system of claim 7, wherein: the instructions toprovide the ranking of the resources comprise instructions to providerepresentations of at least a subset of the resources in ranking order.9. The system of claim 7, wherein: the one or more signals each specifya context in which the respective reported actions occur; and theinstructions to determine a respective LACTR for the respective resourcecomprise instructions to determine a respective LACTR for the respectiveresource in the context based on the reported actions related to therespective resource in the context.
 10. The system of claim 9, whereinthe context is one or more of language, geographic region, or topicalcategory.
 11. The system of claim 7, wherein the one or more signals areembedded in one or more Universal Resource Locators (URLs).
 12. Thesystem of claim 7, the instructions further comprising instructions to:identify first users of the plurality of users, the first usersassociated with an action history that indicates a preference for afirst resource and a second resource of the plurality of resources;identify a second user of the plurality of users, the second usersassociated with an action history that indicates a preference for thefirst resource; and modify a ranking of the second resource with respectto the second user based on the indicated preference for the firstresource by the first and second users.
 13. A computer storage mediumencoded with a computer program, the program comprising instructionsthat when executed by data processing apparatus cause the dataprocessing apparatus to perform operations comprising: for each of aplurality of resources presented to a plurality of users by a dataprocessing service: receiving one or more signals originating from thedata processing service reporting one or more actions by the pluralityof users related to the respective resource, wherein a reported actionis a presentation of the respective resource, an addition of therespective resource, or a deletion of the respective resource, whereinthe reported actions comprise one or more short deletions of therespective resource, a short deletion of the respective resource being adeletion of the respective resource occurring within a predefined timeperiod after a corresponding addition of the respective resource;including: determining a number of long additions for the respectiveresource by subtracting a number of short deletions of the respectiveresource from a number of reported additions of the respective resource;and determining a respective long addition click-through rate (LACTR)for the respective resource by dividing the number of long additions forthe respective resource by a number of reported presentations of therespective resource; ranking the plurality of resources based on therespective LACTRs for the plurality of resources; and providing theranking of the plurality of resources.
 14. The computer storage mediumof claim 13, wherein: providing the ranking of the resources comprisesproviding representations of at least a subset of the resources inranking order.
 15. The computer storage medium of claim 13, wherein: theone or more signals each specify a context in which the respectivereported actions occur; and determining a respective LACTR for therespective resource comprises determining a respective LACTR for therespective resource in the context based on the reported actions relatedto the respective resource in the context.
 16. The computer storagemedium of claim 15, wherein the context is one or more of language,geographic region, or topical category.
 17. The computer storage mediumof claim 13, wherein the one or more signals are embedded in one or moreUniversal Resource Locators (URLs).
 18. The computer storage medium ofclaim 13, the operations further comprising: identifying first users ofthe plurality of users, the first users associated with an actionhistory that indicates a preference for a first resource and a secondresource of the plurality of resources; identifying a second user of theplurality of users, the second users associated with an action historythat indicates a preference for the first resource; and modifying aranking of the second resource with respect to the second user based onthe indicated preference for the first resource by the first and secondusers.